You can picture offhand as something that was never taken in hand first. Nothing was picked up, checked, or held ready before it was used; it simply left the person as it came. Two roads lead out of that picture. One is about where the words came from: nothing was prepared, so the remark is and the answer is . The other is about what the other person feels: because no care went in, they hear someone who could not be bothered with them.
From the word’s history
Wiktionary traces offhand back to Middle English *ofhande and Old English ofhende, meaning absent or lost, from Proto-Germanic *abahandijaz, and says the modern word was re-formed as off- plus hand. So the not-in-hand picture is at least how the word was rebuilt, even though the old form only carried the idea of being away.
維基詞典把 offhand 追到中古英語 *ofhande、古英語 ofhende（不在、失去），再往上到原始日耳曼語，並說現代這個字是照 off- 加 hand 重新組起來的。所以「不在手上」這個畫面，至少符合它被重新拼起來時的想法。但古早那個形式只表示「離開」，沒有現在「沒準備」的意思，這條線當線索就好，不要當成證據。
Wiktionary 'offhand' (kaikki extract): from Middle English *ofhande, *ofhende, from Old English ofhende (absent, lost), from Proto-Germanic *abahandijaz; equivalent to and re-formed as off- + hand.
登入就可以看每一支怎麼連回核心畫面。