Novak Djokovic and Nick Kyrgios were knocked out of the men's doubles in the second round of the Brisbane International by top seeds Michael Venus and Nikola Mektic.

The pair had secured an impressive victory over Alexander Erler and Andreas Mies in the first round, but fell short in the tie-break on Wednesday, losing 6-2 3-6 10-8.

Kyrgios had lost on his singles return on Tuesday, stating after that the defeat had left him in "throbbing" pain, but he showed few signs of discomfort when pairing up with Djokovic once more.

Despite serving six aces and saving three of five break points in the first set, Djokovic and Kyrgios struggled to find their rhythm but improved in the second to force the tie-break.

However, they were two points from victory when the Serbian committed a double fault, which proved costly at 8-6, as the momentum swung back in favour of Venus and Mektic, who then won the final three points in a row to take the match.

"Incredible," Mektic said afterwards. "I was very happy when I saw them [Djokovic and Kyrgios]; I could play them on the first of January.

"It's an amazing feeling to start the year like this, and we knew there would be points like that against players like this."

"Playing those guys, they hit some unbelievable shots," Venus added.

"Down three points in the tiebreak, the way they were serving, it wasn't looking great, but somehow we got back in there."

Meanwhile, in the singles, reigning champion Gregor Dimitrov booked his place in the quarter-finals after easing past Australian wildcard Aleksandar Vukic in straight sets.

The Bulgarian's 6-2 7-6 win was his 25th overall in Brisbane, a tournament he has won twice and finished runner-up at once.

He will face either Jordan Thompson or Alex Michelson for a place in the final four.