The fighting was sparked by a bulldozer crossing a fence and showed how tense the boundary remains nearly six months after a United Nations-brokered ceasefire ended a 34-day conflict between the Israelis and Hezbollah.
UN peacekeepers in south Lebanon described the shootout as a "serious incident" though it was not expected to undermine the truce last year.
Peacekeepers contacted both sides "urging them to cease hostilities with immediate effect."
Lebanese officials said their troops opened fire on an Israeli army bulldozer that had crossed the frontier near the border village of Maroun el-Rass, scene of heavy fighting last summer.
Israel confirmed the exchange with security officials saying Israel's army returned fire with tanks and light weapons but denied its troops had entered Lebanon.
The Israelis said the army was searching for bombs near where explosive devices were allegedly planted by Hezbollah on Monday, meaning Hezbollah had violated the ceasefire.