Tuesday, December 13, 2016

Mo' Better Mocap Shakespeare

The Royal Shakespeare Company's performance-capture production of The Tempestin collaboration with The Imaginarium and Intel, is running a series of talks including one this weekend that I would sorely like to see: Creating The Tempest: Brave New Digital World.

The strapline reads, "What does the future hold for digital storytelling in theatre?" and the speakers include the visionary Sarah Ellis, RSC Head of Digital Development. I had a chance to talk with Sarah this summer about the breathtakingly ambitious production, and all the brain-bending, tech-twisting challenges that go into animating in real time, live on stage.

Mark Quartley as Ariel and Simon Russell Beale as Prospero (photo: Topher McGrillis)

I'm hoping I'll also have a chance to see the final show in person, even if the cinema broadcast makes it to Toronto, because one of those many vexed questions is what surfaces and substances to project the avatar onto once you've cooked it up in the computer.

Judging by the production photos, it looks like the RSC's answer is: "Everything that's nailed down, and maybe a few people who aren't." So it would be a shame to have to squint at that 360-degree experience through the keyhole of a movie screen.

The Company of The Tempest (photo: Topher McGrillis)
But in a way, watching a theatre actor... tracked by infrared motion-capture cameras and inertial sensors... driving real-time digital animation... projected onto a cylindrical scrim and a thrust stage... all recaptured on HD cameras... and broadcast into cinemas around the world? That's no more meta than watching a male actor play the female Rosalind pretending to be the male Ganymede role-playing "Rosalind" so that Orlando can hone his wooing chops.

Simon Russell Beale as Prospero and Mark Quartley as Ariel (photo: Topher McGrillis)

Tuesday, October 4, 2016

Viral Vanier

So on the very same day the Vanier Canada Graduate Scholarships are announced, I learn that not one but two of my PhD classmates in Cinema and Media Studies at York are through to the next round for next year. Go, Claudia and David!

And congratulations, Zachary, Jesse and Syrus Marcus!! Reading about their work, and the research abstracts for the 162 other new Vanier Scholars, feels like looking at one of those gorgeous rainbow MRI scans of Canada's brain...

 Henrietta Howells, NatBrainLab, Wellcome Images (Creative Commons by-nc-nd 4.0)

Tuesday, July 19, 2016

Live Cello Mocap and Projection Mapping

59 Productions have some wicked plans in store for projection mapping in live performance, but here's Sol Gabetta performing Elgar's Cello Concerto in E minor at the BBC Proms, as an example of what they can already do:

(h/t to Liz Barber)

The video below shows the how. It highlights four key challenges with motion capture and projection mapping in live performance.

Saturday, July 9, 2016

Prospero meets Ariel at the Imaginarium

More fascinating glimpses behind the scenes on preparations for the Royal Shakespeare Company's new production of The Tempest: "Simon Russell Beale, playing Prospero, and Mark Quartley, playing Ariel, meet for the first time, coming together with Imaginarium crew and Intel technology to explore how Ariel's avatar will work."

Thursday, July 7, 2016

Performance Capture for Stage, Film and Games

A round-up of links for further exploration of performance capture, offered with respect and gratitude to the massively talented graduating RADA actors who participated in the first-ever Shadowpox workshop – Fehinti Balogun, Natasha Cowley, Sayre Fox, Skye Hallam, Tom MartinPolly MischAbraham PopoolaMaisie Robinson and Jamael Westman.

Tuesday, June 14, 2016

Science + Fiction + Theatre

[This post was originally published in March 2015, but I'm reposting it because it's good background for a workshop I'll be doing next week. As collaborator Simon Eves of PLASTIK pointed out recently, there's still not enough sci-fi onstage...]

At a recent conference on public health history at the University of Toronto, I had some intriguing conversations about crossovers between science and art/entertainment, particularly how science fiction can welcome audiences deep into issues in public health.

It brought back a chat I had last year with Conall Watson of the Department of Infectious Disease Epidemiology at the London School of Hygiene and Tropical Medicine.

He described his work on a promenade theatre event called Deadinburgh, set during a zombie outbreak besieging the Scottish capital.
His team ran the public health advisory cell, "tasked with guiding the audience through different approaches to controlling the zombie epidemic; giving them insight into the usually back-of-house practices of the public health authorities.

We also had input into the epidemiological parameters and narrative of the overall show."

In a paper titled "Deadinburgh: zombie epidemics, citizen power and public health", he and his colleagues Kate Harvey and Nigel Field of University College London, describe the scenario:

"An unknown pathogen was ravaging Scotland’s capital in April 2013, turning unlucky infected souls into bloodthirsty, ambling beasts. The city was under military lock-down and scientists were working around the clock to identify the pathogen and develop means of control.

"Each night, 250 uninfected citizens reached the safe zone at a former veterinary college, taking democratic responsibility for the public health and military response.

"Whether immersive theatre and simulated situations can get people to engage with public health on a larger scale and help build trust and empathy with the way that science is used to inform public sector decision-making remains to be seen. What we do know is that people like science; people like zombies; and the two combined can help us to reflect on our own practice as public health professionals."

In the video below, Kate Harvey says, "Bringing in something from popular culture helps to appeal to a wider audience.... Public health has both art and science at its core. Public health is the art and science of promoting health and preventing disease and prolonging life.... But maybe what we haven't done so much of is using it as a means of communication, and actually putting some of the science back into art as well."

Click here to watch a video of Conall Watson and Kate Harvey discussing "Deadinburgh - the science of zombies" – a London School of Hygiene and Tropical Medicine podcast.

Tuesday, April 12, 2016

So long, green screen

As I work with a Kinect, a projector and a digital video camera to drive and record real-time interactive projected effects, I'm already fantasizing about hacking the Lytro Cinema system together with one o' these.

Come on, it's only a $125,000 rental...

(H/t Jos Humphrey)

The Invisible Pantry

Created for Jennifer Jenson's Cultural Studies of Educational Technology course.

Monday, February 22, 2016

Gamifying Education

Created for Jennifer Jenson's Cultural Studies of Educational Technology course.

(If the presentation embedded below doesn't play within your browser window, you can also view it on the Prezi website.)

Wednesday, February 17, 2016

Live-Animated Simpsons

This just in from the Hollywood Reporter: the May 15th episode of The Simpsons will feature a three-minute segment with Homer Simpson taking questions... live on air. In a genre of animation that normally takes at least six months to produce, this is a 180
Showrunner Al Jean told THR that the series... will use a motion capture technology in which Homer's voice and motions will be depicted in an animated scene talking about things he "could only be saying live on that day."
"HOMЯ", Simpsons episode 9, Season 12
Apparently the segment will look just like the rest of the episode, which means they'll probably be using a 3D model, but toonshading it to flatten the look, which is what we did on Faster than Night and The Augmentalist:

Sunday, February 14, 2016

A Report on Knowledge

(Written as part of Jennifer Jenson's Cultural Studies of Educational Technology course.)

Eight key concepts from Jean-François Lyotard's The Postmodern Condition: A Report on Knowledge (1979):

1. No More Metanarratives

In The Postmodern Condition, Jean-François Lyotard reminds us that the generation and transmission of knowledge is not a frictionless abstraction, but a human activity that costs time, effort and money. In the “modern” era, we justified that cost through a belief that knowledge brought us closer to something even larger: the “Emancipation of the people,” say, or a unified Idea of truth and justice. Yet Lyotard argues that we have entered a new era in which we no longer put stock in such “grand narratives”, an era he therefore dubs “postmodern”.

“I will use the term modern to designate any science that legitimates itself with reference to a metadiscourse of this kind making an explicit appeal to some grand narrative, such as the dialectics of Spirit, the hermeneutics of meaning, the emancipation of the rational or working subject, or the creation of wealth… Simplifying to the extreme, I define postmodern as incredulity toward metanarratives.” (p.xxiii)

2. Commercialization of Knowledge

Where once the concept of education embraced a broad and fuzzy process of intellectual maturation, enculturation and even spiritual growth, Lyotard fears knowledge is being reduced to an economic good. We increasingly weigh the “use-value” of knowledge (its intrinsic value to its holder and to society) less than its “exchange-value”, in the form of an individual worker’s marketable skills (“payment knowledge”), or of knowledge enabling an organization or nation to improve its efficiency and future well-being (“investment knowledge”). Lyotard also sees this commoditization as setting the stage for conflict between the traditional State and the new multinational corporation over the control of information.

“We may thus expect a thorough exteriorisation of knowledge with respect to the ‘knower,’ at whatever point he or she may occupy in the knowledge process. The old principle that the acquisition of knowledge is indissociable from the training (Bildung) of minds, or even of individuals, is becoming obsolete and will become ever more so. The relationships of the suppliers and users of knowledge to the knowledge they supply and use is now tending, and will increasingly tend, to assume the form already taken by the relationship of commodity producers and consumers to the commodities they produce and consume – that is, the form of value.” (p.4-5)

3. Language Games

Lyotard specifies that what he means by knowledge is not simply learning, science, teaching or research, although it embraces all of these activities. Knowledge gives us the ability to go beyond mere regurgitation of facts or assertion of truth (“denotative utterances”), to venture assertions of justice (“prescriptive utterances”), of efficiency and even of beauty (“evaluative utterances”). These utterances are very different kinds of “moves” in “language games” with very different rules, any one of which can be used to justify or legitimate the aforementioned learning, science, teaching or research.

I distinguished the denotative game (in which what is relevant is the true/false distinction) from the prescriptive game (in which the just/unjust distinction pertains) from the technical game (in which the criterion is the efficient/inefficient distinction). ‘Force’ appears to belong exclusively to the last game, the game of technology.” (p.46)

4. Performativity vs. Research

Lyotard identifies performativity (or efficiency, i.e. greater output for less input) as the most popular justification in capitalist society. For example, what he calls the “research game”, which requires funding because it is not inevitably or immediately profitable in itself, is nevertheless required by its state or corporate funders to provide evidence of its long-term contribution to the improvement of efficiency and the growth of profit or power. Lyotard believes this choice of justification has a chilling effect on the advancement of knowledge, creating a “terrorist” system in which researchers avoid bold moves for fear of being removed from the game.

“Research funds are allocated by States, corporations, and nationalized companies in accordance with this logic of power growth. Research sectors that are unable to argue that they contribute even indirectly to the optimization of the system’s performance are abandoned by the flow of capital and doomed to senescence.” (p.47)
5. Paralogy is the Answer

Lyotard’s distaste for totalizing metanarratives leads him to conclude that a better justification for knowledge is the pursuit not of performativity, but paralogy, the multiplication of ideas, “little narratives”, perspectives and possibilities. He stresses that paralogy is not simple innovation, a goal-oriented and profit-oriented activity, nor is it a multi-vocal conversation aimed at eventual consensus, but rather the never-ending generation and dissension of stories and ideas, many of which will have no immediately obvious value.

“[T]he little narrative remains the quintessential form of imaginative invention, most particularly in science. (p.60)

6. Will Tech Replace Teachers?

Landing squarely on the number-one fear (or hope) raised by an examination of knowledge in computerized societies, Lyotard starts with the assertion that a computer might be able to take over the more didactic parts of a teacher’s duties. However, education is not just about imparting information or professional skills-training, but also stimulating imagination and invention, something a human is better suited to than a machine. And yet Lyotard points out that the latter type of learning need not take place in a university at all, so perhaps the question is not whether teachers will be replaced by technology, but by teamwork.

“[T]he process of delegitimization and the predominance of the performance criterion are sounding the knell of the age of the Professor: a professor is no more competent than memory bank networks in transmitting established knowledge, no more competent than interdisciplinary teams in imagining new moves or new games.” (p.53)

7. Temporary Contracts, Ongoing Education

Whoever is doing the teaching, Lyotard sees the societal trend toward short-term contracts as changing the timeframe of education. Whereas once people received all their training in one extended early period, now they are more likely to need retraining at several points during their working life (given the flexibility demanded by the late-capitalist economy, quite a number of points). Yet this new model of education is likely to privilege workforce training over the formation of well-rounded, thoughtful citizens.

“In the context of delegitimation, universities and the institutions of higher learning are called upon to create skills and no longer ideals … The transmission of knowledge is no longer designed to train an elite capable of guiding the nation towards its emancipation, but to supply the system with players capable of acceptably fulfilling their roles at the pragmatic posts required by its institutions.” (p48)
8. Information Ought to be Free

At the end of his investigation Lyotard raises the spectre of a future society where computers enable the complete privileging of the performative criterion, of calculation and control. Over a decade before the world wide web, though, he also imagined how open-source information-sharing could power an infinite variety of human endeavour.

“[T]he computerization of society… could also aid groups discussing metaprescriptives by supplying them with the information they usually lack for making knowledgeable decisions… [by giving] the public free access to the memory and data banks.” (p.67)

How might some of Lyotard's ideas still be useful for understanding education/learning/the world today?

Lyotard’s lenses on the various “values” of knowledge are highly relevant to the ongoing debate on funding for higher education. A grasp of the concepts of “exchange-value” and “use-value” makes it easier to contextualize arguments that the job market should dictate what the nation pays for its youth to learn, or that all academic research should feed back into industry. 

That said, for all Lyotard’s belief that we have moved beyond grand narratives, neoliberalism has created its own all-devouring metanarrative of market efficiency, and if paralogy hopes to make a counter-move in the language game of education, it needs to tell a more compelling story.

We are also living squarely in the age, not of IBM satellite belts orbiting the earth, but of Mother Google providing everything from Google Apps for primary students, to the perpetual “continuing education” of the average adult’s search-engine reflex to any question – all for “free”. While it’s impossible to know how paranoid is too paranoid in such a world, we should never stop asking Lyotard’s questions, “Who will have access to them? Who will determine which channels or data are forbidden?... Who will know?”

Finally, on a more personal note, I found Lyotard's concept of the antagonistic relationship between scientific knowledge and narrative knowledge fascinating from the perspective of the vaccination debate. The resentment of those who feel their narratives (for example, the notion that vaccines cause autism) have been unfairly dismissed by scientists as unsubstantiated fables was cast in the new light of “the entire history of cultural imperialism from the dawn of Western civilization.” That doesn’t make such narratives any more scientifically valid, but it is a useful perspective on their history and appeal.

Sunday, February 7, 2016

What is twenty-first-century magic?

More from the team behind the Royal Shakespeare Company's performance-capture Tempest:
"What's twenty-first-century magic? Well, I guess it comes in some kind of digital form these days. We've started to see a lot more in film, in the world of movies, but we've never really explored that in the theatre context.... It's the creative process that drives the technology, and not the other way around. So it started to be a match made in heaven, from that point of view." 
– Stephen Brimson Lewis, Director of Design, RSC

"With The Tempest we're really trying to redefine theatre, in some respects, and find a way to bring in new digital technology and really leverage it to make the story deeper, to find new ways to connect with a character, and maybe a different audience... an entirely new generation."
– Tawny Schlieski, Research Scientist, Intel

"The most exciting thing about what we're doing at the moment is enabling an actor to have a real connection with an avatar. It's the sort of thing that I do on a daily basis in the studio for films and video games. In this instance, we're doing something that's going to be a live theatrical experience. What you see is what you get." 
– Ben Lumsden, Head of Studio, The Imaginarium

"The play demands a spectacle. There's a masque in the middle of the play, an insubstantial pageant, which fades into nothing, but it creates wonder. I want to let the guys at Intel know a bit about what that tradition was, just how elaborate those masques were, and how they were pushing the envelope." 
– Gregory Doran, Artistic Director, RSC 

Modern Foundations of Literacy: Ibn Rushd

(Written as part of Jennifer Jenson's Cultural Studies of Educational Technology course.)

ابن رشد  
Ibn Rushd

ʾAbū l-Walīd Muḥammad Ibn ʾAḥmad Ibn Rušd, known in the West as Averroes (via the Hebrew “Aben Rosh”), was born in 1126 in Córdoba, Spain, where he was educated in Islamic jurisprudence – his father and grandfather were both chief justices in the city.

He moved to Marrakesh, Morocco, and gained the patronage of the caliph 'Abd al-Mu'min, who was interested in educational reform. This was a problematic venture, however, as the government’s liberalizing movement was opposed by anti-philosophical theologians, and this increasing tension eventually led to Ibn Rushd’s banishment and the burning of his books. He was allowed to return to Marrakesh after two years, but died a year later in 1198.

فلسفة | Philosophy

Rationalism. Quoting verse 59.2 of the Qur’an, "Reflect, you have vision," Ibn Rushd believed that education should equip (some) people to enquire and reason rather than blindly accepting through faith. He argued that rational philosophy and the scientific method could explain natural phenomena, contradicting the theologians who asserted that natural phenomena could be attributed directly and immediately to God’s will. Note, however, that Ibn Rushd was not secular in the modern sense (and would have suffered worse than banishment had he been), but rather held that the laws that controlled nature had been designed and set in motion by God.

علم النفس | Psychology

The soul is divided into five faculties: the nutritive, the sensitive, the imaginative, the appetitive and the rational. The rational faculty, which gives humans the ability to create, understand, and act ethically, moves from potentiality to actuality through a process called “intellectation”. The potential/material intellect passively receives forms through the senses, and thus activated, it grows through stages of habit and acquired thought, and eventually, through an ever-greater contemplation the eternal and the universal, the soul itself becomes eternal and universal.

علم الإجتماع | Sociology

Ibn Rushd agrees with Plato in The Republic that women should be educated, and regrets that Islam relegates them to childbearing. He believes that women’s participation in the economy, military, academy and government would be of great benefit to society. Ibn Rushd takes Plato to task, however, in not providing for the happiness of all levels of society. He believes Islamic Shari’ah is superior to Plato’s Republic in providing a variety of ways to know God and find happiness: not just the inner meaning which can only be interpreted by the metaphysician, but also the metaphor which is accessible to the uneducated believer.

مفهوم محو الأمية | Conception of literacy

For Ibn Rushd, there is no conflict between religion and philosophy – both are ways to the knowledge of truth, but each form of "literacy" is suited to a different kind of person. Philosophy, which requires more training and intellectual capacity, should be reserved for the elite. Religion, based in faith and expressed through metaphoric imagery, requires no special training if taken at face value, and is therefore well-suited to the masses. (As a tool to encourage the masses to behave ethically, he sees the Judeo-Christian focus on the spiritual rewards of heaven as inferior to the Muslim emphasis on the afterlife’s physical pleasures. But while he appreciates the artistry of this promise, he himself finds it hard to credit.)

الموقف في التعليم | Attitude to education

Spiritual value. Training in philosophy enables the (elite) believer to understand God’s laws at a deeper level.

منهاج دراسي | Curriculum

Ibn Rushd earned the nickname “The Commentator” for his popularizing editions of Aristotle and other ancient philosophers, which were designed to make these texts more accessible to his contemporaries. Latin translations of his works were largely responsible for the rediscovery of these texts by the West in the 12th and 13th centuries, and while much of the Arab world grew more skeptical that society needed to read anything beyond the Qur’an and the hadith, Ibn Rushd eventually came to be considered "founding father of secular thought in Western Europe". He also wrote a medical text titled Generalities (general medicine), and invited his friend Ibn Zuhr to write a companion text, Particularities. Together these became the main medical textbooks for centuries in the Muslim and Western worlds.

التربية | Pedagogy

Ibn Rushd distinguished between “the people of rhetoric” and “the people of demonstration”. The former, the vast majority, will take simple and superficial teachings at face value. The latter are able to follow complex philosophical reasoning, and need to be allowed to question more deeply. Overhearing these questionings would, however, be harmful the masses, who might misunderstand and fall into dangerous unbelief, so philosophical inquiry must be kept a guarded secret.

تقييم | Evaluation

The son of the caliph, Abu Yaqub Yusuf, was well-educated but still complained that he found the Aristotle disjointed and obscure. He commissioned Ibn Rushd to write summaries and commentaries of the Greek philosophers to help people understand them better. If I were to speculate on Ibn Rushd’s examination techniques, which do not come down to us through the centuries, I might imagine him requiring his students to write their own original commentaries on ancient texts.

النتيجة | Outcome

Knowledge of Truth / knowledge of God.

الأفكار حول التعليم القرن ال21 | Thoughts about 21st century education

Ibn Rushd would be happy to see women more included in 21st century education. He might shake his head at policies that aim at providing the same curriculum for all students regardless of intellectual ability and moral agility. The unanswerable question is whether he would be relieved by the secular nature of many modern education systems, or would be dismayed by the removal of God from the pursuit of knowledge. As it is impossible to tell at this distance whether his own religious faith was a political obligation or something deeply felt, we must leave this question open.

فهرس | References

Thursday, January 14, 2016

Capturing Coldplay

The making of Coldplay and The Imaginarium's Adventure of a Lifetime video – one of the best behind-the-scenes on performance capture I've ever seen, and definitely the most fun!

And here's the original video: Coldplay – Adventure of a Lifetime.

(H/t to Neil Richards)

Wednesday, January 13, 2016

So this goes on the Christmas list...

New prototype high-speed projection mapping system – at CES 2016, Panasonic will unveil its projector technology for high-speed and synchronized mapping on unmarked target objects:

"Prototype." I hope that doesn't mean Christmas 2018...

Monday, January 11, 2016

"Stratford upon Avatar"

(Sorry, couldn't resist nicking the Daily Telegraph's headline.)

Very exciting announcement today by the RSC:
"Today the Royal Shakespeare Company announced a new production of William Shakespeare’s late play The Tempest, produced in collaboration with Intel and in association with leading performance capture company, The Imaginarium Studios. The companies will combine their passion for storytelling and innovation to create a truly revolutionary production as part of the RSC’s winter season 2016.

Directed by RSC Artistic Director, Gregory Doran, with RSC Associate Artist, Simon Russell Beale, as Prospero, and designed by Stephen Brimson Lewis, this partnership will see the RSC’s skills at theatre-making come together with Intel’s digital innovation and the expertise of The Imaginarium in pushing technical boundaries to create a truly innovative production for a new generation to mark Shakespeare’s 400th anniversary.

For the very first time, performance capture technology will be used to render an animated character - Ariel the sprite - live on the Royal Shakespeare Theatre stage.

The technology works by capturing an actor’s facial expressions as well as their movements, which ensures that an actor’s full performance is translated into the animated character. It has most famously been used in films and gaming, but together the RSC, Intel and The Imaginarium have undertaken more than a year of research to bring digital avatars to life on stage in real-time, interacting with live actors."
Intel expands:
"Today the Royal Shakespeare Company announced a new production of William Shakespeare's play The Tempest, produced in collaboration with Intel and in association with The Imaginarium Studios. The companies will collaborate to create a revolutionary production as part of RSC's winter season 2016. For the first time, performance capture technology will be used to render an animated character live on the Royal Shakespeare Theatre stage. Standard Intel technology is used to manage massive data processing required for live digital content projection - from Intel Xeon to Intel Core i7 processors."
I heard some tantalizing murmurs about this plan last summer, and can't wait to see what they and The Imaginarium cook up together!

The Telegraph's report is the most extensive so far: