Father James discusses why he became a priest with Lincoln. He says that he lied previously about why, and that he didn't become one because of what he saw in the war, he became one because of what he did there and what it turned him into. He explains that while he was over there, crammed together with all the other "disposable negroes" that all the hurt and rage he'd accumulated over his life came boiling up.
All the men he killed in the war, should have meant something to him, they should have had weight, but it didn't. When he glimpsed that part of himself it was too much. He's worried that Lincoln isn't feeling the weight of what he's doing and it terrifies him.