Building a better A&S model

[lumineaux]

Having been dragged kicking and screaming into the "arts and sciences" end of the SCA world, I've observed some things, which competing in King's and Queen's A&S Champions only confirmed.

Best intentions aside, we do not do a good job of giving artisans and "scientists" informed feedback and critiques.  This problem is particularly acute for people who work outside of the very common fields of garb-making, C&I, embroidery and (perhaps) cooking.

The problem is that people can only judge and give informed commentary about fields where they already have some knowledge.  I can discourse lovingly on heraldry, period heraldic style from certain areas and eras, some aspects of C&I, period music and a few other areas, but ask me whether someone's embroidery technique is either good or period and I am lost.  

We cannot always guarantee that a judge or judges with knowledge will be available for a competition, particularly for works outside of the mainstream.  Without informed feedback, how can people working outside of the usual arts ever improve?

Because I think like a Pelican, my first thought is about how to fix the problem. How do we educate judges about how to judge things  they aren't familiar with?   Being someone with a very limited area of specialization myself, I'd love to get a crash course from someone about things to look for when evaluating works I'm not personally familiar with.


Would it make sense to have "primers" from acknowledged "experts" (in so far as we have any) about what to look for when judging a certain topic?  Even that is going to have some biases, and could detract from innovation and new ways of looking at data.

Thoughts?  Opinions?  A general consensus that I am full of shit?

Comments

[lanome.livejournal.com]

Would it make sense to have "primers" from acknowledged "experts" (in so far as we have any) about what to look for when judging a certain topic? Even that is going to have some biases, and could detract from innovation and new ways of looking at data.

I think that might be helpful, but I think we need to take a few steps before we can get there. I think we could probably start with classes on:

How to judge an A&S entry (even when you are familiar with the subject).
How to give feedback that is meaningful, but not hurtful.
How to calibrate scores for different arts against each other.
How to read documentation.
What to bring with you when judging A&S competitions.

[zaduzbina.livejournal.com]

I am looking at it from the entrant's perspective. How do "We" process how we awere judged?

[lumineaux.livejournal.com]

Ah, but can you really compare the hours of data extraction and analysis it takes to prepare a paper on Scottish names and naming patterns in the 13th century with the hours of research and technical skills needed to create a sugar sculpture subtlety in any meaningful way?

One of the reasons why I never liked having A&S "Champions" is that I don't think you can meaningfully pit one against the other. All Rapier Champions got there by fighting Rapier. All Archery Champions by shooting. All Heavy List Champions by beating each other senseless with sticks.

I'd rather see us do more encouraging and promoting the arts and sciences in a period way -- tokens from the Crown, thanks, displays, etc. -- than "competing" for a Champion position. But I also understand that people love their cookies and a competition will push people into finishing projects that otherwise would linger.

[alysten.livejournal.com]

"How to read documentation."

That one is tricky. But if there were some guidelines for judges that might help. Things that you look for in cooking are very different from things you look for in sewing. Where things get muddy are things to look for in heraldry, pottery, pick any A&S that is outside of the regular type things. Page count limitations also makes it very difficult to communicate information about a piece. Judges may be left with a spotty or incomplete picture.

[lanome.livejournal.com]

On further reflection, I might have to do some more A&S judging so I can teach some of these classes. Although, if anyone is willing to teach them in the mean time, I would like to take them.

Wow. Great minds.....

[zaduzbina.livejournal.com]

I am actually ranting on this elsewhere. And have begun a small group where we are talking about this very issue.

We should collaborate:

My soon to be article:

After the Ball. How to process what happened to you as an entrant in a competition.

[baronessmartha.livejournal.com]

I would love to be in on this if that is okay with you.

I entered Ice Dragon last year. It was the first time in 15 years I entered any A&S competition. It was a very positive experience and I would be happy to share why it was positive.

[math5.livejournal.com]

My soon to be article:

Please make sure to send it to the Ars Scientia Orientalis

[hlwoods.livejournal.com]

I would love to read (and possibly join) this discussion if I may. I have only competed three years, but done reasonably well in the process. And even then, some judges' comments leave me stumped.

[bds-duncan.livejournal.com]

I recall a discussion with a Laurel who related that she had been approached to judge a competition in a field far outside her own area of knowledge and expertise, for the person asking her to judge had presumed that as a Laurel this individual was automatically competent to judge anything. Sadly, this is not the case, and most Laurels know this. Yet is is difficult to find competent judges for every possible category of art and science. Therefore we stumble along with a broken system that provides little or no useful feedback to the artisan.

I honestly have problems with A&S competitions, as they too often seem to pit different disciplines against one another with totally different criteria. How does one honestly compare blackwork with a culinary creation or pottery with a research paper? Perhaps there is a judging scheme that can provide a level playing field - I do not know. But as they are an almost necessary evil, seeking competent judges in more varied fields would be a first response. Educating judges and potential judges on how best to give constructive feedback would be a second response. Encouraging potential judges to widen their fields of interest (and thus widen the pool of potential judges in a given area) would be a third.

[goldsquare.livejournal.com]

You are never full of shit.

If I were entering A&S, and there was documentation for the Judges on "how to Judge", as an entrant I'd read the heck out of that. It's like crib notes, you know?

I've seen this sort of thing discussed generally before, and I always see a long disputation on the various reasons that people enter for comment, and another on how various people's personality types react to various formats of feedback.

You're working in the more rarefied air of heraldic expertise here - you should probably consider submitting these things to the experts, like Saint Gabriels.

[lumineaux.livejournal.com]

I've already discussed my articles and research with other name heralds and gotten their far more valuable feedback. Entering K&Q A&S was an academic exercise on my part to see what would happen.

[lumineaux.livejournal.com]

You are never full of shit

Oh, I am quite frequently full of shit, to varying degrees. ;-)

[ursula]

Does the East do face-to-face A&S judging?

[lumineaux.livejournal.com]

What do you mean by face-to-face judging?

I submitted research papers and scroll texts, all of which were reviewed in advance and not the day of the event, so I wouldn't have gotten "face to face" feedback regardless. I was teaching heraldry stuff all day, so I don't know whether any of the other people displaying or competing had a chance to talk with the judges in person or not.

[math5.livejournal.com]

In some competitions, others are check entries and documentation.

[baronessmartha.livejournal.com]

It would make some sense, what also makes sense is for judges to try to work together (I have seen them do it very well), and for entrants to make their documentation tell a story. The second part is tricky to know unless someone tells you.

I was asked to judge a&s at a tiny tiny event as I was a peer. When I said "I am a pelican, it is a different thing entirely" they said "please".

I looked at things as best I could and judged on things like how much the entrant taught me about something, how much effort they put into it. Judging is hard, yo.

[lumineaux.livejournal.com]

I got caught in a similar situation where the autocrat wanted to have a heraldic display competition. I agreed to judge, but I think no one was entirely happy with the results, because I judged based on what I know: good period style and period methods of heraldic display. So I didn't care if the person made the prettiest garb if the heraldry displayed on it and the manner of display weren't period. It turns out that wasn't quite what the entrants were expecting.

[lissa510.livejournal.com]

" judged on things like how much the entrant taught me about something"

this is the same thing that I heard from a Laurel about how they judge topics they do not know.

[alysten.livejournal.com]

A&S this weekend was a lot like Rapier K&Q round robin. Churn through the pieces and judging. There were just so many different things that needed to be looked at. I understand the desire to have a well rounded artisan, but there were 90 different things that had to be judged by 20 different people.

I am a big fan of oral critiques. If there were a bit more communication between judges and competitor, that would help the lack of written commentary. They don't take that much more time (well they could if you have a "chatty" judge) and I have found I've gotten better feedback than the written sheets.

[bunnyjadwiga.livejournal.com]

posty mc-comment-a-lot here. One of the difficulties with A&S comps is that as a volunteer organization, we can't depend on a) knowing what people will enter ahead of time and b) being able to be reasonably sure someone will be able to make it to judge that subject.

[lissa510.livejournal.com]

I really think that the best place for people to compete and recieve feedback would be in a guild dedicated to their art or art "genre"

[math5.livejournal.com]

We don't have guilds for all the A&S people do, yet. But there's progress :)

[oaken-glen.livejournal.com]

Well, I've never entered a competition. And I've not gotten to the point where I can actual *do* what I want to do (unless I rent time in a shop somewhere). I want to enter letterpress stuff at some point. However, the person I consider to be *the expert* on all things re: hand set type moved to Colorado. Sure, there are folks in the Kingdom that do bookbinding, paper making, etc. But there's only a handful of people (that I know of) that is branching out into wood block printing and letterpress work.

So...how can and will that be judged? Dunno. It'll be interesting to find out though...

[ursula]

How do we educate judges about how to judge things they aren't familiar with?

This is what good documentation is for! Documentation should tell the judge what was done in period and what process the contestant followed, in enough detail that the judge can evaluate whether the contestant's extrapolations are reasonable.

I should say that documentation can be a mix of writing, pictures, and verbal explanations. I personally prefer systems where judges can talk to entrants.

Documentation may not tell you how original someone is being, in terms of whether they're re-inventing a wheel that lots of folks in the SCA have studied, or trying something totally different. And it may not tell you whether something is technically challenging. Disciplinary expertise helps here.

[shaunacarrick.livejournal.com]

Speaking from afar:

In Artemisia, we have had good results with the 'how to judge' classes and the 'how to write up documentation' classes being given at multiple events prior to the competition events.

I have been a judge for many different categories - some of them I know better than others, certainly. One time for a Kingdom competition (we only have one a year, generally), I was asked to help judge the brewing. I asked how many entries there were. I was told there were five competitors, so I said 'sure'. I thought that would mean only five items to taste.

Wrong.

Five entrants who entered multiple items in multiple sub-categories in the brewing 'grand category'. Myself and the two other judges had to take at least one break each to eat and sober up! I think there was a total of 30 different cordials - no beers, all hard liquors! - and it was a very good thing that they had people to double-check all our math for our individual score sheets!

Anyway, I am always glad to judge, here, and look forward to being educated by the documentation for areas or items I am not familiar with.

[mariedeblois.livejournal.com]

I'm gonna be Commenty McCommentPants when I get the chance, but I wanted to start by saying "I hear ya, sister".

This all sounds a lot like the things I was thinking about A&S about a few years ago when I was entering/sponsoring people Kingdom A&S (champions competition) and Queen's Prize (display and everybody-gets-feedback-and-a-prize-from-a-sponsor). I might, however, have been a bit more inclined towards "crack the system wide open so people can see the broken things inside". ;) You can see some ideas in the sadly abandoned LJ community aspradigmshift. I *know* I wrote some posts to my LJ at the time, but now I can't find em.