I used to listen to ALOT of avantgarde stuff.
Maudlin of the Well
Not really metal, but they have their moments especially from their album Bath and Leaving Your Body Map has some extremely brutal death metal and doom. They have a surreal feel to their music, almost Dr Seuss-like. They went on to rename themselves Kayo Dot which was more post-rock/freeform jazz/ambient. But Maudlin of the Well just recently released another album available for download on their website called Part the Second, masterpiece..but not metal.
Solefald
"Post-black" metal band that utilizes saxophone, catchy riffs and harmonizing vocals. Almost very similar to Arcturus.
Sigh
Everyone should know this Japanese band, their album Imaginary Sonicscape is amazing and weird and probably the least thrashy or black metal than their other albums, alot of jazz/big band interludes and cheesy sounding melodies that somehow fits the strange feel of the album. Mirai knows how to compose some really weird sounding songs.
The Meads of Asphodel
Fucking weird, takes the piss out of bands that prance around in weapons and armor. They have a punk edge and combines stupid sounding techno with some black metal riffs. Not my favourite, good for a laugh.
Estradasphere
Fucking amazingly talented songwriters. These guys know how to be extremely schizophrenic with their song writing, fusing genres of death, big band, doom, surf, carnival/circus, black metal, nintendo, comedy, blues yet still maintaining an extremely coherent sound. One of my favourites, check out their album Buck Fever.
If you're a fan of Augury's first album, check out UneXpect. Very similar styles.
Naked City
John Zorn's grindcore/free jazz/noise project. Interesting listen.
Ephel Duath
Check out everything they've done, they've had some great musicians in this band. My personal favourite album is The Painter's Palette.
I'm sure there's more in my collection but I can't remember as I'm not at home currently, I'll sift through tomorrow during the day and post some more probably!