OK, minor setback. I discovered some smoke and felt the smell of burned electronics, but could not find out why at first. Then I realized that the circuit drew out the batteries WAY to fast. It consumed close to 10 mA which is too much. At last I discovered that it seems to be the small LED-driver on the spotlight board that really did not like that I removed the LED:s. I cut the power and the input signal to it (the two pins at the blue arrow) by scraping off the copper leads with a knife.
Now the power consumption of that board is down to 500uA which is healthier for the batteries.