Explanation

"From" is used to show removal or separation (taking something away).

So, "I had to take my book from the table" is correct.