The windows I installed need to be caulked. There is caulk that I think is still good on the table in the aux room along with a gun and some useful tools of mine, as shown below. The intent is not just to seal, but the caulk provides a bit of cushion between the glass and the frame. The tolerances are smaller than I intended, so getting caulk down in between the frame and glass may take a bit of effort. Both sides of both windows need to be done eventually, but feel free to do just part if you want.
Door and window caulk is usually white, which will match the adjacent border here and match with what we used on the other windows. I have some to spare from personal work that I will bring in. (Not that the color is critical at all.)