Genesis 37:36

37:36 Now in Egypt the Midianites sold Joseph to Potiphar, one of Pharaoh’s officials, the captain of the guard.

Genesis 40:4

40:4 The captain of the guard appointed Joseph to be their attendant, and he served them.

They spent some time in custody.