Superintelligence: Science or Fiction?

A noteworthy conversation!

Elon Musk, Stuart Russell, Ray Kurzweil, Demis Hassabis, Sam Harris, Nick Bostrom, David Chalmers, Bart Selman, and Jaan Tallinn discuss with Max Tegmark (moderator) what likely outcomes might be if we succeed in building human-level AGI, and also what we would like to happen.

The Beneficial AI 2017 Conference: In our sequel to the 2015 Puerto Rico AI conference, we brought together an amazing group of AI researchers from academia and industry, and thought leaders in economics, law, ethics, and philosophy for five days dedicated to beneficial AI. We hosted a two-day workshop for our grant recipients and followed that with a 2.5-day conference, in which people from various AI-related fields hashed out opportunities and challenges related to the future of AI and steps we can take to ensure that the technology is beneficial.

Posted in Blog | Leave a comment

AI Aftermath Scenarios

From Life 3.0 : being human in the age of artificial intelligence by Max Tegmark

Libertarian utopia

Humans, cyborgs, uploads and superintelligences coexist peacefully thanks to property rights.

Benevolent dictator

Everybody knows that the AI runs society and enforces strict rules, but most people view this as a good thing.

Egalitarian utopia

Humans, cyborgs and uploads coexist peacefully thanks to property abolition and guaranteed income.


A superintelligent AI is created with the goal of interfering as little as necessary to prevent the creation of another superintelligence. As a result, helper robots with slightly subhuman intelligence abound, and human-machine cyborgs exist, but technological progress is forever stymied.

Protector god

Essentially omniscient and omnipotent AI maximizes human happiness by intervening only in ways that preserve our feeling of control of our own destiny and hides well enough that many humans even doubt the AI’s existence.

Enslaved god

A superintelligent AI is confined by humans, who use it to produce unimaginable technology and wealth that can be used for good or bad depending on the human controllers.


AI takes control, decides that humans are a threat/nuisance/waste of resources, and gets rid of us by a method that we don’t even understand.


AIs replace humans, but give us a graceful exit, making us view them as our worthy descendants, much as parents feel happy and proud to have a child who’s smarter than them, who learns from them and then accomplishes what they could only dream of—even if they can’t live to see it all.


An omnipotent AI keeps some humans around, who feel treated like zoo animals and lament their fate.


Technological progress toward superintelligence is permanently curtailed not by an AI but by a humanled Orwellian surveillance state where certain kinds of AI research are banned.


Technological progress toward superintelligence is prevented by reverting to a pre-technological society in the style of the Amish.


Superintelligence is never created because humanity drives itself extinct by other means (say nuclear and/or biotech mayhem fueled by climate crisis).

Posted in Blog | Leave a comment

How Important is Software?

From Life 3.0 : being human in the age of artificial intelligence by Max Tegmark

"On June 4, 1996, scientists hoping to research Earth’s magnetosphere cheered jubilantly as an Ariane 5 rocket from the European Space Agency roared into the sky with the scientific instruments they’d built. Thirty-seven seconds later, their smiles vanished as the rocket exploded in a fireworks display costing hundreds of millions of dollars. The cause was found to be buggy software manipulating a number that was too large to fit into the 16 bits allocated for it. Two years later, NASA’s Mars Climate Orbiter accidentally entered the Red Planet’s atmosphere and disintegrated because two different parts of the software used different units for force, causing a 445% error in the rocket-engine thrust control. This was NASA’s second super-expensive bug: their Mariner 1 mission to Venus exploded after launch from Cape Canaveral on July 22, 1962, after the flight-control software was foiled by an incorrect punctuation mark. As if to show that not only westerners had mastered the art of launching bugs into space, the Soviet Phobos 1 mission failed on September 2, 1988. This was the heaviest interplanetary spacecraft ever launched, with the spectacular goal of deploying a lander on Mars’ moon Phobos—all thwarted when a missing hyphen caused the “end-of-mission” command to be sent to the spacecraft while it was en route to Mars, shutting down all of its systems."

Posted in Blog | Leave a comment

Behave: the biology of humans

Conclusions of Behave: the biology of humans at our best and worst by Robert M. Sapolsky.


· It’s great if your frontal cortex lets you avoid temptation, allowing you to do the harder, better thing. But it’s usually more effective if doing that better thing has become so automatic that it isn’t hard. And it’s often easiest to avoid temptation with distraction and reappraisal rather than willpower.

· While it’s cool that there’s so much plasticity in the brain, it’s no surprise—it has to work that way.

· Childhood adversity can scar everything from our DNA to our cultures, and effects can be lifelong, even multigenerational. However, more adverse consequences can be reversed than used to be thought. But the longer you wait to intervene, the harder it will be.

· Brains and cultures coevolve.

· Things that seem morally obvious and intuitive now weren’t necessarily so in the past; many started with nonconforming reasoning.

· Repeatedly, biological factors (e.g., hormones) don’t so much cause a behavior as modulate and sensitize, lowering thresholds for environmental stimuli to cause it.

· Cognition and affect always interact. What’s interesting is when one dominates.

· Genes have different effects in different environments; a hormone can make you nicer or crummier, depending on your values; we haven’t evolved to be “selfish” or “altruistic” or anything else—we’ve evolved to be particular ways in particular settings. Context, context, context.

· Biologically, intense love and intense hate aren’t opposites. The opposite of each is indifference.

· Adolescence shows us that the most interesting part of the brain evolved to be shaped minimally by genes and maximally by experience; that’s how we learn—context, context, context.

· Arbitrary boundaries on continua can be helpful. But never forget that they are arbitrary.

· Often we’re more about the anticipation and pursuit of pleasure than about the experience of it.

· You can’t understand aggression without understanding fear (and what the amygdala has to do with both).

· Genes aren’t about inevitabilities; they’re about potentials and vulnerabilities. And they don’t determine anything on their own. Gene/environment interactions are everywhere. Evolution is most consequential when altering regulation of genes, rather than genes themselves.

· We implicitly divide the world into Us and Them, and prefer the former. We are easily manipulated, even subliminally and within seconds, as to who counts as each.

· We aren’t chimps, and we aren’t bonobos. We’re not a classic pair-bonding species or a tournament species. We’ve evolved to be somewhere in between in these and other categories that are clear-cut in other animals. It makes us a much more malleable and resilient species. It also makes our social lives much more confusing and messy, filled with imperfection and wrong turns.

· The homunculus has no clothes.

· While traditional nomadic hunter-gatherer life over hundreds of thousands of years might have been a little on the boring side, it certainly wasn’t ceaselessly bloody. In the years since most humans abandoned a hunter-gatherer lifestyle, we’ve obviously invented many things. One of the most interesting and challenging is social systems where we can be surrounded by strangers and can act anonymously.

· Saying a biological system works “well” is a value-free assessment; it can take discipline, hard work, and willpower to accomplish either something wondrous or something appalling. “Doing the right thing” is always context dependent.

· Many of our best moments of morality and compassion have roots far deeper and older than being mere products of human civilization.

· Be dubious about someone who suggests that other types of people are like little crawly, infectious things.

· When humans invented socioeconomic status, they invented a way to subordinate like nothing that hierarchical primates had ever seen before.

· “Me” versus “us” (being prosocial within your group) is easier than “us” versus “them” (prosociality between groups).

· It’s not great if someone believes it’s okay for people to do some horrible, damaging act. But more of the world’s misery arises from people who, of course, oppose that horrible act . . . but cite some particular circumstances that should make them exceptions. The road to hell is paved with rationalization.

· The certainty with which we act now might seem ghastly not only to future generations but to our future selves as well.

· Neither the capacity for fancy, rarefied moral reasoning nor for feeling great empathy necessarily translates into actually doing something difficult, brave, and compassionate.

· People kill and are willing to be killed for symbolic sacred values. Negotiations can make peace with Them; understanding and respecting the intensity of their sacred values can make lasting peace.

· We are constantly being shaped by seemingly irrelevant stimuli, subliminal information, and internal forces we don’t know a thing about.

· Our worst behaviors, ones we condemn and punish, are the products of our biology. But don’t forget that the same applies to our best behaviors.

· Individuals no more exceptional than the rest of us provide stunning examples of our finest moments as humans.

Posted in Blog | Leave a comment


Quote from Behave: the biology of humans at our best and worst by Robert M. Sapolsky (my bullets and highlights)

Evolution rests on three steps: (a) certain biological traits are inherited by genetic means; (b) mutations and gene recombination produce variation in those traits; (c) some of those variants confer more “fitness” than others. Given those conditions, over time the frequency of more “fit” gene variants increases in a population.

We start by trashing some common misconceptions.

First, that evolution favors survival of the fittest. Instead evolution is about reproduction, passing on copies of genes. An organism living centuries but not reproducing is evolutionarily invisible.* The difference between survival and reproduction is shown with “antagonistic pleiotropy,” referring to traits that increase reproductive fitness early in life yet decrease life span. For example, primates’ prostates have high metabolic rates, enhancing sperm motility. Upside: enhanced fertility; downside: increased risk of prostate cancer. Antagonistic pleiotropy occurs dramatically in salmon, who epically journey to their spawning grounds to reproduce and then die. If evolution were about survival rather than passing on copies of genes, there’d be no antagonistic pleiotropy.

Another misconception is that evolution can select for preadaptations—neutral traits that prove useful in the future. This doesn’t happen; selection is for traits pertinent to the present. Related to this is the misconception that living species are somehow better adapted than extinct species. Instead, the latter were just as well adapted, until environmental conditions changed sufficiently to do them in; the same awaits us. Finally, there’s the misconception that evolution directionally selects for greater complexity. Yes, if once there were only single-celled organisms and there are multicellular ones now, average complexity has increased. Nonetheless, evolution doesn’t necessarily select for greater complexity—just consider bacteria decimating humans with some plague.

The final misconception is that evolution is “just a theory.” I will boldly assume that readers who have gotten this far believe in evolution. Opponents inevitably bring up that irritating canard that evolution is unproven, because (following an unuseful convention in the field) it is a “theory” (like, say, germ theory). Evidence for the reality of evolution includes:

· Numerous examples where changing selective pressures have changed gene frequencies in populations within generations (e.g., bacteria evolving antibiotic resistance). Moreover, there are also examples (mostly insects, given their short generation times) of a species in the process of splitting into two.

· Voluminous fossil evidence of intermediate forms in numerous taxonomic lineages.

· Molecular evidence. We share ~98 percent of our genes with the other apes, ~96 percent with monkeys, ~75 percent with dogs, ~20 percent with fruit flies. This indicates that our last common ancestor with other apes lived more recently than our last common ancestor with monkeys, and so on.

· Geographic evidence. To use Richard Dawkins’s suggestion for dealing with a fundamentalist insisting that all species emerged in their current forms from Noah’s ark—how come all thirty-seven species of lemurs that made landfall on Mt. Ararat in the Armenian highlands hiked over to Madagascar, none dying and leaving fossils in transit?

· Unintelligent design—oddities explained only by evolution. Why do whales and dolphins have vestigial leg bones? Because they descend from a four-legged terrestrial mammal. Why should we have arrector pili muscles in our skin that produce thoroughly useless gooseflesh? Because of our recent speciation from other apes whose arrector pili muscles were attached to hair, and whose hair stands up during emotional arousal.

Posted in Blog | Leave a comment

Constituția Sexului în România

(Publicat inițial pe Strada Democrației pe 26 iunie 2017)

Pe 25 iunie 2017 s-a încheiat Pride Month în Toronto, ce a culminat cu anualul Pride March. Este pentru a doua oară în istoria Canadei când un șef de guvern participă la un astfel de marș, după ce anul trecut prim-ministrul Justin Trudeau a fost primul șef de guvern canadian care a participat la Pride March. Alături de el a participat și premierul provinciei Ontario, Kathleen Wynne, care e declarat lesbiană.

Atitudinea mea față de acest eveniment? Aprob, încurajez, suport. Nu m-am amestecat niciodată în astfel de mulțimi, nu pentru că nu m-ar fi mânat un soi de curiozitate ludică, ci pentru că le simt ca pe niște manifestări normale, democratice și nu le disting de Caribana, carnavalul caraib ce atrage un milion de turiști anual, sau de Festivalul de Film din Toronto (TIFF). Toate aceste înfăptuiri de evenimente sînt, într-un fel sau altul, manifestări de cultură.

Context istoric: activitățile între persoane de același sex sînt legale în Canada din 1969, persoanele LGBTQ+ pot presta serviciu militar din 1992, căsătoria între persoanele LGBTQ+ a fost legalizată în 2005. Drepturile persoanelor LGBTQ+ în Canada sînt printre cele mai avansate din lume. Pierre Trudeau (prim-ministru între 1980-1984), tatăl lui Justin Trudeau, a declarat în 1967 că “Statul nu are ce căuta în dormitoarele poporului“.

Să punem apoi în balanță Coaliția pentru Familie. Ce vrea Coaliția? Coaliția vrea refuz, izolare, discriminare, limitare de drepturi, excludere și întoarcerea la valori tradiționale. Valorile tradiționale sînt definite, desigur, pe criterii religioase. E mai mult decât regretabil că aceste valori nu se revendică de la tradiția iluminismului, tradiție care la rândul ei ne-a dat statele moderne și încetățenirea metodei științifice de investigare a cunoașterii. Din tradiția iluminismului se revendică și revoluțiile moderne ale drepturilor civice.

Coaliția persistă să se hrănească cu o tocăniță de meme foarte fragile. În paragrafele de mai jos voi oferi niște metode de deconstrucție a acestor meme. Mică paranteză: conceptul de memă a fost inventat de Richard Dawkins în Gena Egoistă (1976). Pe scurt idea lui Dawkins e că mema imită strategia genei. Gena e unitatea fundamentală de reproducere în contextul evoluției prin selecție naturală. Organismul biologic e doar un vehicul folosit de genă pentru a-și executa misiunea ei programată în ADN. Pe de altă parte, mema e un fel de genă alcătuită din idei, iar organismul e alcătuit din societate și cultură.

Memele Coaliției pentru Familie sînt:

  • Actele sexuale non-hetero sînt abominabile și constituie un mare păcat
  • Familia e alcătuită din bărbat și femeie
  • Cupluri LGBTQ+ nu pot avea copii, deci contrazic dictatul divin de procreare a speciei
  • Persoanele LGBTQ+ sînt de o moralitate inferioară și prin urmare nu pot fi considere egale cu societatea heterosexuală
  • Societatea fără valori creștine e o societate în derivă și sortită pierzaniei
  • Preceptele de mai sus au atribute de lege organică și moralitate absolută deci trebuie introduse în constituție

Bill Nye, în episodul 9 al serialului Bill Nye Saves the World, deconstruiește dualitatea sexuală de bărbat versus femeie și prezintă în schimb un spectru de gen non-binar de la bărbat la femeie, pe patru criterii: sex, gen, atracție și expresie. SEXUL e biologic și e determinat de organele genitale, de cromozomi și de hormoni. GENUL e cum individul se identifică pe sine și cum se auto-percepe din punct de vedere sexual. Sînt oameni a căror SEX și GEN nu se află de aceeași parte a spectrului. De exemplu: persoanele transgender pot avea un creier de bărbat într-un corp de femeie, sau vice-versa. Alte persoane se identifică ca ambele genuri, sau niciunul sau o combinație de amândouă. ATRACȚIA se referă la obiectul atracției fizice. Persoanele homosexuale sînt atrase de sexul opus, persoanele bisexuale sînt atrase de ambele sexe, persoanele asexuale de nici un sex. Atracția e la rândul ei așezată pe un spectru. EXPRESIA e cum se prezintă individul societății. Aici se includ: cine poartă machiaj, cine se îmbracă în fuste, când și unde, cine se rade pe picioare, cine se rade între picioare, cine e drag queen și cine nu.

Memele Coaliției sînt fragile tocmai pentru că sînt binare și extrem de rigide argumentativ. O campanie media puternică de educare a societății pornind de la instrumentarul de mai sus ar demonta rapid elanul militant al Coaliției. Cazul împotriva memelor Coaliției trebuie făcut metodic, puternic, perseverent, folosind:

  • Instrumentarul de mai sus cu multe exemple din țările progresiste (vezi Canada)
  • Puterea juridică a drepturilor omului
  • Demontarea argumentului religios folosind tehnici de relativizare post-modernă. Narațiunile ortodoxe, prin faptul că fac apel la un vocabular arhaic pot fi demontate printr-un contra-vocabular modern dar simplu și ușor de digerat de votanții petiției Coaliției.

Desigur, progresul moral al societății nu este inevitabil. După Imperiul Roman am avut Evul Mediu întunecat. După iluminism, am avut războaie mondiale, gulag-uri și genocid. Deși în măsura în care unele grupări încearcă să înmlăștinească conceptul de familie, uitând de pildă că grecii antici practicau homosexualitatea fără să aibă conceptul de sex și gen, iar unii preoți creștini întrec starurile porno în perversiuni, există totuși cale de speranță și progres când spiritul civic al progresiștilor se lansează în proiecte sociale și politice.

Posted in Blog | 1 Comment

The price of everything

"Every day we commit to buying goods and services without paying careful attention to their cost. In 2009, the HP DeskJet D2530 printer might have seemed a steal at $39.99. But the price, displayed prominently on the HP Web site, was almost irrelevant. The more relevant numbers were $14.99 for a black ink cartridge, which prints about 200 pages, and $19.99 for the color cartridge, which prints 165. For those printing photos at home, the crucial number was $21.99 for the HP 60 Photo Value Pack, a set of cartridges and 50 standard sheets of photo paper. At the Rite-Aid drugstore, 50 same-day prints cost $9.50. The worldwide printing business depends on selling cheap printers and expensive ink. According to a study by PC World, printers will issue out-of-ink warnings when the cartridge is still up to 40 percent full. HP, Epson, Canon, and others have sued providers of cheap ink refills, charging them with false advertising and patent infringement to make them stop. But the best ally of the printer business is consumer ignorance about what they are really paying to print. Just setting the printer default to “draft” quality would save consumers hundreds of dollars a year. Yet few consumers do. Though many companies still sell cheaper ink refills, refills account for only 10 to 15 percent of the market. That means that 90 percent of printing is still done using ink that, according to the PC World analysis, costs $4,731 per gallon. You might as well fill your ink cartridges with 1985 vintage Krug champagne."

(from The price of everything : solving the mystery of why we pay what we do / Eduardo Porter)

Posted in Blog | Leave a comment