JACKSON (AP) — A federal appeals panel has agreed with a lower court judge that Noxubee County Democratic chairman Ike Brown violated white voters’ rights in an eastern Mississippi county.

A three-judge panel of the 5th Circuit U.S. Court of Appeals in New Orleans upheld a 2007 ruling by U.S. District Judge Tom S. Lee, siding with the Justice Department against Brown. The panel’s decision came Friday.

In a rare case that used the 1965 Voting Rights Act to assert that white voters were disenfranchised by a black official, the Justice Department had accused Brown of trying to limit whites’ participation in local elections.

The 5th Circuit panel agreed with Lee, saying Brown and the local Democratic Executive Committee ‘‘engaged in a series of episodic practices — in obtaining a large number of black absentee ballots, in preventing the proper challenges to absentee ballots, and in permitting numerous instances of improper voter assistance — that diluted the votes of white Democrats.’’

Brown, reached Monday by cell phone, denied that he tried to limit anyone’s rights. Brown said he was proud of his efforts to get more blacks to vote, whether by absentee ballot or getting them to the polls.

‘‘Did Judge Lee rule I kept anybody from voting? No,’’ Brown told The Associated Press. ‘‘Did Judge Lee rule that I kept anybody from running? No. Judge Lee said because there were so many blacks voting absentee, it must be fraudulent. How do you defend against that?

‘‘Nobody has brought charges of absentee voting fraud in Noxubee County.’’

Brown said he would consider an appeal to the U.S. Supreme Court.

A Justice Department spokesman had no immediate comment.

In the 2007 decision, Lee ruled that Brown and the Noxubee County Democratic Executive Committee ‘‘manipulated the political process in ways specifically intended and designed to impair and impede participation of white voters and to dilute their votes.’’

The case was a civil matter carrying no criminal penalties, but defendants who violate Lee’s final order could face contempt of court charges and fines, prosecutors said.

Noxubee County is 69 percent black and 29 percent white, according to the U.S. Census.

Lee said there was a pattern to Brown’s efforts to keep all whites out of the county’s Democratic Party. It included holding party caucuses in private homes rather than public voting precincts and inviting only blacks to the meetings.

Brown had argued whites lose elections in Noxubee County because most voters prefer black candidates. However, Lee said evidence showed Brown had a race-based agenda to keep whites from winning elections.

The 5th Circuit panel said the Voting Rights Act does not guarantee minority success at the polls but does require that minorities participate in the political processes and have a chance to elect representatives of their choice.

Brown and the executive committee, the panel said, wanted the court ‘‘to stay its hand and ’let things evolve’ in Noxubee County because they pursued what they view as an honorable goal of electing black candidates.’’

‘‘To do so, however, defendants have wielded their authority over the election process to violate Mississippi’s election laws at the expense of Mississippi voters.

‘‘To champion the cause of black candidates by abusing the supervisory authority over elections in order to undermine the value of white voters’ ballots breaches the boundary between acceptable political activity and unacceptable electoral abuse,’’ wrote panel members Carolyn Dineen King, James L. Dennis and Jennifer Walker Elrod.

The panel also rejected Brown’s challenge to Lee’s requirement that Brown remove himself from any official duties involving the 2007 countywide election and the appointment of former Mississippi Supreme Court Justice Reuben Anderson to oversee all Democratic primaries and runoffs through Nov. 20, 2011, in Noxubee County.