From a slow start Ellsworth built up a prosperous law practice. His reputation as an able and industrious jurist grew, and in 1777 Ellsworth became Connecticut's state attorney for Hartford County. That same year he was chosen as one of Connecticut's representatives in the Continental Congress. He served on various committees during six annual terms until 1783. Ellsworth was also active in his state's efforts during the Revolution. As a member of the Committee of the Pay Table, Oliver Ellsworth was one of the five men who supervised Connecticut's war expenditures. In 1779 he assumed greater duties as a member of the council of safety, which, with the governor, controlled all military measures for the state.

When the Constitutional Convention met in Philadelphia in 1787 Ellsworth once again represented Connecticut and took an active part in the proceedings. During debate on the Great Compromise, Ellsworth proposed that the basis of representation in the legislative branch remain by state, as under the Articles of Confederation. He also left his mark through an amendment to change the word "national" to "United States" in a resolution. Thereafter, "United States" was the title used in the convention to designate the government.

Ellsworth also served on the Committee of Five that prepared the first draft of the Constitution. Ellsworth favored the three-fifths compromise on the enumeration of slaves but opposed the abolition of the foreign slave trade. Though he left the convention near the end of August and did not sign the final document, he urged its adoption upon his return to Connecticut and wrote the Letters of a Landholder to promote its ratification.

Ellsworth served as one of Connecticut's first two senators in the new federal government between 1789 and 1796. In the Senate he chaired the committee that framed the bill organizing the federal judiciary and helped to work out the practical details necessary to run a new government. Ellsworth's other achievements in Congress included framing the measure that admitted North Carolina to the Union, devising the non-intercourse act that forced Rhode Island to join, drawing up the bill to regulate the consular service, and serving on the committee that considered Alexander Hamilton's plan for funding the national debt and for incorporating the Bank of the United States.

In the spring of 1796 he was appointed Chief Justice of the Supreme Court and also served as commissioner to France in 1799 and 1800. Upon his return to America in early 1801, Ellsworth retired from public life and lived in Windsor, CT. He died there on November 26, 1807, and was buried in the cemetery of the First Church of Windsor.